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FOREWORD 


The  Instruction  Manual  contains  detailed  instructions  for  evaluation 
With  the  Index  of  Electronic  Equipment  Operability.  The  manual  is  one  of 
five  related  documents.  In  addition  to  the  manual,  the  Data  Store  and 
Evaluation  Booklet  are  required  for  evaluating  equipment.  The  Sample 
Equipment  Evaluations  report  contains  detailed  evaluations  of  four  equip¬ 
ments,  including  recommendations.  The  Report  of  Development  constitutes 
the  final  technical  report,  and  describes  the  development  of  the  index. 

It  is  appropriate  here  to  call  the  attention  of  Index  users  to  the 
interpretation  of  Index  results.  Because  the  results  of  evaluation  are 
numerical  in  nature,  there  may  be  a  recurrent  tendency  to  overemphasize 
the  results.  It  must  be  pointed  out  that  these  results  are  meaningful 
only  when  interpreted  within  the  context  of  all  that  is  known  about  the 
equipment.  Divorced  from  this  context,  the  results  may  be  misleading. 
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INTRODUCTION 


This  manual  is  intended  to  provide  guidance  for  evaluators  in  apply¬ 
ing  the  Index  of  Electronic  Equipment  Operability.  In  addition  to  this 
manual,  application  of  the  Index  requires  a  Data  Store  booklet  and  an 
Evaluation  Booklet.  The  Data  Store  provides  quantitative  standards  for 
estimating  human  reliability  and  speed  of  performance  for  various  types 
of  operations.  The  Evaluation  Booklet  contains  a  standard  format  for 
recording  the  results  of  an  evaluation.  Background  information  con¬ 
cerning  the  Index  and  a  description  of  its  development  are  presented  in 
the  technical  report,  Development  of  an  Index  of  Electronic  Equipment 
Operab i 1 i ty . 


This  instruction  manual  is  divided  into  three  major  sections.  The 
first  section  provides  a  general  description  of  the  Index  and  its  pur¬ 
pose.  The  second  section  contains  detailed  instruct'ons  for  applying 
the  Index.  Explicit  instructions  are  provided  for  the  data  collection, 
scoring,  and  summarizing  steps  in  the  evaluation.  General  guidance  is 
offered  for  the  judgmental  processes  of  interpreting  the  results  of  the 
evaluation  and  deriving  design  and  other  recommendations.  The  third  and 
final  section  of  the  manual  is  the  Appendix,  which  contains  samples  of 
the  various  forms  used  in  the  application  of  the  Index. 


DESCRIPTION  OF  THE  INDEX 


Purpose 

The  purposes  of  the  Index  of  Electronic  Equipment  Operability  are 
to: 

1.  Predict  the  average  time  required  for,  and  reliability 
of  operator  performance. 

2.  Provide  a  quantitative  basis  for  identifying  specific 
human  engineering  design  problems  and  developing  recom¬ 
mendations  for  overcoming  or  minimizing  these  problems. 

3.  Organize  the  results  of  the  evaluation  in  a  form  which 
facilitates  due  consideration  for  selection  and  train¬ 
ing  requirements. 

Assumptions 

To  assure  that  the  Index  achieves  thr  purposes  stated  above,  the 
following  conditions  must  be  met  in  each  application  of  the  Index, 

1.  Available  equipment  and  task  information  must  accurately 
describe  the  design  and  operating  character i st ics  of  the 
equipment  to  be  evaluated.  Any  change  in  the  design  of 
the  equipment  or  the  allocation  of  operator  responsibili¬ 
ties  will  alter  the  detailed  evaluation  results  and  may 
significantly  alter  the  interpretation  of  the  results. 

2.  The  Index  is  intended  for  prediction  of  performance  by 
relatively  unselected  personnel  who  have  received  only 
nominal  training.  In  most  cases,  rigid  selection  criteria 
or  intensive  training  will  result  in  operator  performance 
that  is  faster  and  more  reliable  than  performance  predicted 
by  the  Index. 

3.  The  Index  should  be  applied  by  a  professional  human  factors 
engineer  or  other  personnel  qualified  to  evaluate  man- 
machine  interactions. 
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Information  Required  for  Evaluation 


Two  general  types  of  information  concerning  the  equipment  to  be 
evaluated  must  be  obtained  before  the  Index  can  be  successfully  applied 
to  the  equipment. 

1.  Equipment  Information.  Data  concerning  the  equipment 
should  include  layout  of  the  equipment,  where  and  how  it 
will  be  installed,  the  mission  it  is  intended  to  accom¬ 
plish,  reaction  time  in  meeting  mission  objectives,  and 
detailed  information  about  the  displays  and  controls 
available  to  the  operator.  Potential  sources  of  such 
information  include  equipment  descriptions  and  design 
drawings  prepared  during  the  development  of  the  equip¬ 
ment,  mock-ups,  and  operational  and  maintenance  manuals. 

2.  Operating  Information.  The  second  type  of  required  in¬ 
formation  for  the  evaluation  should  contain  a  detailed 
description  of  the  activities  required  of  the  operator 
during  normal  and  emergency  modes  of  operation.  This 
information  may  be  obtained  from  a  task  analysis,  opera¬ 
tional  manuals,  or  directly  from  a  proficient  operator 
of  the  equipment.  Even  with  multiple  sources  of  infor¬ 
mation  concerning  operation  of  the  equipment,  it  may 
still  be  necessary  fo^  the  evaluator  to  depend  upon 
inferences  about  operation,  particularly  with  regard 

to  the  mental  processes  required  of  the  operator. 

For  both  types  of  information  specified  above,  it  will  be  necessary  for 
the  evaluator  to  cross-check  the  validity  of  the  data  obtained  to  insure 
that  the  description  of  the  equipment  and  its  operation  is  completely 
accurate.  In  all  cases,  multiple  sources  of  information  should  be  util¬ 
ized  if  avai lable. 

In  general,  the  later  the  stage  of  equipment  development  at  which 
the  evaluation  is  performed,  the  more  complete  and  accurate  will  be  the 
information  available  to  the  evaluator. 
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J)  The  basic  problem  in  evaluating  human  behavior,  such  as  that  required 
of  an  equipment  operator,  is  that  complex  behavior  is  seldom  duplicated 
from  one  situation  to  another  in  complete  detail.  It  is  necessary,  there¬ 
fore,  in  order  to  analyze  and  evaluate  such  behavior,  to  first  reduce  it 
to  a  small  enough  unit  of  behavior  that  it  may  be  compared  with  available 
information  on  similar  microscopic  units  from  a  number  of  situations.  For 
purposes  of  the  Operability  Index,  a  behavior  is  considered  to  be  a  dis¬ 
crete  step  or  action  in  a  given  cask.  Placing  the  high  voltage  switch  in 
the  ON  position  would,  for  example,  be  one  step  in  a  task  required  for 
the  activation  of  an  equipment  unit. 

In  order  to  establish  estimates  of  time  and  reliability  in  performing 
a  given  behavior  or  step,  it  is  first  analyzed  into  the  following  three 
aspects: 

1.  Input  or  stimulus  which  initiates  the  behavioral  step. 

The  stimulus  may  come  from  an  indicator  such  as  a  light, 
scope,  or  meter;  from  an  informational  job  aid  such  as  a 
manual  or  checklist;  from  what  the  operator  "knows"  must 

-  be  done;  or  from  successful  completion  of  the  preceding 
step. 

2.  Mediating  process  or  thinking  which  is  required  between 
receiving  the  input  and  making  the  proper  response.  This 
mediating  process  may  be  virtually  automatic,  fairly  simple, 
such  as  comparing  two  numerical  inputs,  or  quite  complex, 
such  as  evaluating  threats. 

3.  Output  or  response  which  is  the  overt,  observable  part  of 
the  behavior  and  may  involve  such  things  as  verbal  communi¬ 
cation,  writing,  or  activating  a  control. 

The  Index  is  based  on  the  independent  assessment  of  the  performance 
time  and  operator  reliability  associated  with  each  of  these  three  aspects 
of  input,  mediating  process,  and  output  for  each  behavioral  step.  Time 
and  reliability  are  a  function  of  certain  characteristics  of  each  of  the 
three  aspects  of  behavior.  Relevant  categories  of  character  ist  ics  for 
each  aspect  of  behavior  are  presented  in  the  Data  Store,  along  with  time 


and  error  estimates  attributable  to  each.  The  first  major  step  of  the 
evaluation,  therefore,  is  determining  the  characteristics  relevant  to  a 
particular  step  of  behavior  being  analyzed  and  then  matching  these  charac¬ 
teristics  with  those  contained  in  the  Data  Store.  A  sample  page  from  the 
Data  Store  is  presented  as  Figure  1. 


JOYSTICK 

(May  move  in  many  planes) 


BASE  TIME  * 

1-93 

Time  added 

Reliability 

1 

.9963 

1  .50 

0 

•  9967 

1 .50 

.9963 

2 

0 

.9981 

.20 

.9975 

.50 

.9960 

3. 

0 

.9999 

.50 

.9992 

4. 

0 

.9990 

1.00 

.9950 

5. 

0 

.9967 

.50 

•  9963 

3.00 

.9957 

Stick  length 

a.  6-9" 

b.  12-18" 

c.  21-27" 

Extent  of  stick  movement  (Extent 
of  movement  from  one  extreme  to 
the  other  in  a  single  plane.) 

a.  5-20° 

b.  30-40° 

c.  40-60° 

Control  resistance 

a.  5-10  lbs. 

b.  10-30  lbs. 

Support  of  operating  member 

a.  Present 

b.  Absent 

Time  delay  (Time  lag  between  move 
ment  of  control  and  movement  of 
display.) 

a.  .3  sec, 

b.  .6-1.5  sec. 

c.  3.0  sec. 


Figure  1.  Sample  Data  Store  Card 


The  example  presented  in  f  gure  I  relates  to  the  output  aspect  of  be¬ 
havior  associated  with  the  move  ient  of  a  joystick  control.  Entry  to  this 
information  in  the  Data  Store  would  be  accomplished  by  first  identifying 
that  the  output  aspect  of  behavior  being  analyzed  was  through  movement  of 
a  joystick.  Joystick  then  is  the  component  level  of  analysis  Further 
analysis  of  the  behavior  would  reveal  the  relevant  parameters,  such  as 
stick  length,  control  resistance,  etc.  for  the  particular  situation. 
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Finally,  the  relevant  d imens ions  of  the  behavior,  such  as  the  actual  length 
of  the  stick  could  be  determined.  It  is  at  this  level,  dimensions,  that 
matching  of  the  behavior  being  analyzed  with  the  content  of  the  Data  Store 
occurs.  In  most  cases,  more  than  one  parameter  will  be  relevant  to  the 
aspect  of  behavior  being  analyzed.  In  these  cases,  the  times  and  relia¬ 
bilities  associated  with  the  various  dimension  values  for  the  parameters 
concerned  are  combined.  This  combination  is  a  simple  addition  for  time 
values  and  a  mu  1 1 ip  I icat ion  of  reliabilities. 

A  potential  source  of  confusion  in  conducting  the  behavior  analysis 
may  be  the  role  played  by  feedback.  Feedback  may  be  defined  as  the  infor¬ 
mation  resulting  from  operator  performance;  for  example,  the  "feel"  of  a 
control  as  it  goes  into  a  detent  position.  The  following  two  types  of  feed 
back  are  assessed  in  different  ways  for  the  purposes  of  the  evaluation. 

1.  Directing  feedback.  The  first  type  of  information  result¬ 
ing  from  operator  performance  serves  only  to  direct  or  aid 
control  manipulation.  Stimuli  or  information  that  primarily 
aid  control  manipulation,  such  as  labels  or  associated  dis¬ 
play  features,  are  assessed  as  parameters  of  that  control. 

Data  relating  to  this  type  of  feedback  is  contained  in  the 
Data  Store  for  control  components  under  the  parameter . 

"Clarity  of  Control  Indications." 

2,  Initiating  feedback.  Feedback  which  signals  the  end  of  one 
step  of  behavior  and  serves  as  the  input  to  the  next  step, 
such  as  an  indicator  light  or  scale  value,  is  assessed  with 
the  step  of  behavior  it  initiates.  Such  feedback  may  be 
ignored,  however,  if  there  are  other  inputs  which  initiate 
the  behavior  or  If  initiating  feedback  does  not  have  a 
clear  role  in  initiating  the  subsequent  behavior. 


Between  Stec 


In  some  situations,  operational  time  is  consumed  between  the  end  of 
one  behavioral  step  and  the  beginning  of  the  next.  This  time  is  usually  a 
result  of  either  required  perceptual  or  location  shift  by  the  operator. 
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Perceptual  shift  refers  to  the  time  required  for  the  operator  to  shift 
his  attention  from  one  control  or  display  to  another.  Perceptual  shift 
does  not  require  that  the  operator  physically  move  from  one  location  to 
another.  Normally,  the  time  required  for  perceptual  shift  can  be  ignored, 
since  it  is  of  very  short  duration.  It  should  be  noted  in  the  evaluation 
only  when  excessive.  Location  shift  refers  to  a  physical  movement  of  the 
operator  from  one  station  to  another  and  includes  perceptual  shift  as  a 
component  of  the  required  behavior.  Physical'  shift  would  normally  be 
recorded  in  the  evaluation,  since  it  is  usually  of  longer  duration  than 
a  simple  perceptual  shift. 

In  addition  to  loss  of  time  due  to  perceptual  or  location  shift  by 
the  operator,  operational  time  may  be  consumed  by  requirements  of  the  equip¬ 
ment  itself.  This  equipment  delay  time  may  be  due  to  such  design  features 
of  the  equipment  as  warm-up  time,  circu!try  lags,  etc.  Loss  of  time  due 
to  equipment  delay  should  be  recorded  to  provide  a  realistic  estimate  of 
totalmission  time  required  for  the  equipment. 


Evaluation  Results 

The  results  of  the  behavior  and  between  step  analysis  previously  des¬ 
cribed  are  recorded  on  the  Evaluation  Sheet  and  totals  are  computed  for 
each  aspect  of  behavior  for  the  step  and  for  the  step  as  a  whole.  After 
this  worksheet  is  completed,  summaries  are  prepared  based  on  the  step  and 
aspect  values. 

The  first  summary  prepared  is  the  computation  of  totals  for  each  phase 
of  the  mission  and  for  the  total  mission.  These  totals  are  entered  on  the 
Mission  and  Phase  Summary  Form.  Results  are  separately  summarized  on  this 
form  for  the  behavioral  aspects  of  input,  mediating  process,  and  output, 
as  well  as  for  perceptual  shift,  location  shift,  and  equipment  delay. 

The  second  summary  prepared  contains  totals  by  component  for  each  of 
the  mission  phases  and  is  reported  on  the  Component  Summary  Form. 

The  purpose  of  the  summary  forms,  samples  of  which  are  contained  in 
the  Appendix  of  this  report,  is  to  present  in  concise,  usable  form  the 
information  required  by  the  evaluator  in  developing  systematic  recommen¬ 
dations  dealing  with  the  design  of  the  equipment,  training  for  operation 
of  the  equipment,  and  selection  of  personnel  to  operate  the  equipment. 
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Summary  Description 


A  graphical  summary  of  the  basic  evaluation  process  Involved  in  the 
Index  is  presented  In  Figure  2.  Essentially,  the  Individual  steps  of 
operation  are  analyzed  in  their  component  parts.  Scores  for  these  com¬ 
ponents  are  determined  with  the  aid  of  the  Data  Store.  The  component 
scores,  and  between  step  time  scores,  are  then  combined  into  step  scores. 
The  step  scores  can  then  be  combined  in  various  ways  to  yield  total  as¬ 
pect,  phase,  and  mission  scores.  Total  scores  for  specific  components 
are  taken  from  the  general  component  scores.  This  array  of  quantitative 
information  of  different  levels  can  then  be  used  to  guide  decisions  and 
recommendations  concerning  the  equipment  evaluated. 
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Figure  Z.  Graphical  S unary  cf  the  Basic  Evaluation  Process 


INSTRUCTIONS 


The  application  of  the  Index  requires  the  completion  of  six  major 
steps  or  processes.  These  steps  are  listed  briefly  below.  Detailed 
instructions  for  each  step  are  contained  in  subsequent  sections  of  the 
report . 

1 .  Organize  Equipment  and  Operating  Information.  Data 
obtained  from  task  analyses  and  other  sources  must  be 
analyzed  into  behavioral  steps  and  sequenced  by  mission 
phases  of  operation. 

2.  Collect  Evaluation  Data.  This  step  includes  the  identi¬ 
fication  of  relevant  components,  parameters,  and  dimen¬ 
sions  for  each  step;  matching  these  values  with  the  data 
in  the  Data  Store;  and  entering  the  appropriate  values 
on  the  Evaluation  Sheet. 

3.  Score  Evaluation  Sheet.  Step  scores  are  computed  for 
each  aspect  of  behavior  and  across  aspects  for  total 
step  scores  by  adding  together  the  relevant  time  entries 
and  multiplying  together  the  reliability  estimates.  These 
totals  are  entered  on  the  Evaluation  Sheet. 

4.  Summarize  Results  by  Mission  and  Phase.  Total  values  for 
each  phase  of  the  mission  and  for  the  tota1  mission  are 
computed  from  the  data  on  the  Evaluation  Sheet  similar 

to  the  method  for  obtaining  step  totals.  The  results 
of  this  summary  are  entered  on  the  Mission  and  Phase  Sum¬ 
mary  sheet. 

5.  Summarize  Results  by  Component.  Total  values  for  each 
component  of  the  input,  mediating  process,  and  output 
aspects  of  behavior  are  computed  across  the  steps  of  each 
phase  of  the  mission.  The  values  are  entered  on  the  Com¬ 
ponent  Summary  Form. 

6.  Derive  Recommendations.  Based  on  the  summarized  results 
of  the  evaluation  listed  above,  recommendations  may  be 
developed  in  the  following  three  areas: 
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a.  Redes i qn .  Redesign  recommendations  are  based  on 
consideration  of  total  component  scores  on  the  Com¬ 
ponent  Summary  Form  and  selection  of  alternate 
dimensions  from  the  information  contained  in  the 
Data  Store  to  improve  potential  operator  performance 

b.  Training.  Training  recommendations  will  be  based 
on  analysis  of  the  Component  Summary  Form  and  will 
identify  aspects  of  performance  that  should  be 
given  special  attention  in  the  training  of  operators 

c.  Selection.  Selection  recommendations  will  be  based 
on  identification  of  aspects  of  behavior  which  con¬ 
tribute  significantly  to  total  mission  scores  on  the 
Mission  and  Phase  Summary  Form.  These  aspects  may 
then  be  related  to  general  selection  requirements 
for  operators  emphasizing  these  aspects. 


ORGANIZE  EQUIPMENT  ANO  OPERATING  INFORMATION 


1.  Determine  the  basic  mission  of  the  system  based  on  an  analysis 
of  the  documents  cited  previously,  such  as  manuals,  task  analy 
ses,  etc.  The  mission  of  a  system  might  be,  for  example,  "to 
detect  and  locate  enemy  troop  movements." 

2.  Analyze  the  mission  of  the  system  into  the  operational  phases 
required  to  accomplish  the  mission.  The  initial  phase  of  a 
system  might  be,  for  example,  activation  of  the  system,  or 
preliminary  adjustment  of  the  system, 

3.  Order  the  operator  behaviors  sequentially  in  each  phase  of 
the  mission.  The  listing  of  behaviors  should  be  checked  to 
insure  that  they  involve  only  0  single  step  or  action,  since 
that  is  the  level  required  for  the  analysis. 

4.  In  the  event  that  there  are  alternate  procedures  or  sequences 
of  behavior  available  to  the  operator,  include  only  the  most 
common  or  most  frequent  in  the  evaluation. 

5.  If  the  operation  of  the  equipment  in  a  particular  phase  re¬ 
quires  the  repetition  of  a  behavioral  step,  this  step  should 
be  included  in  the  evaluation  as  often  as  it  is  likely  to 
occur.  This  repetition  situation  is  likely  to  occur  with 
certain  alignments  or  adjustments  of  the  equipment. 

6.  Behavior*  which  are  required  only  if  some  relatively  unlikely 
event  takes  place  should  be  assessed  only  if  this  unlikely 
event  is  an  integral  part  of  the  system  mission. 

7.  The  final  list  of  sequenced  behavioral  steps  for  each  phase 
should  then  be  recorded  in  some  convenient  and  appropriate 
format.  No  specific  format  is  specified  in  the  evaluation. 
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COLLECT  EVmLUATION  DATA 


Analyse  the  first  step  of  the  first 
mission  pn.ise  and  enter  the  step 
number  on  the  Evaluation  Sheet.  (A 
complete  Evaluation  Sheet  is  included 
as  page  1  of  the  Appendix.) 

Identify  the  source  of  the  Input  in 
equipment  nomenclature  and  record  it. 

Locate  the  relevant  Input  component 
in  the  Data  Store  and  enter  it  on 
the  form. 

Enter  the  Base  Time  associated  with 
the  component. 

Using  the  Data  Store  as  a  checklist, 
determine  the  parameters  relevant  to 
the  particular  situation.  The  para¬ 
meters  selected  from  the  Data  Store 
must  be  matched  ,ith  the  pre-printed 
n  umber  sunder  the  P  a  r  J '  __(F  a  ramie  t  e  r ) 
coljmn.  For  each  relevant  parameter, 
the  dimensions  which  best  describe 
tne  attributes  of  this  particular 
component  should  be  noted,  oy  code 
letter,  under  the  column  headed 
"Dim.  Ind."  (Dimension  Indication). 
Some  parameters  in  the  Data  Store 
for  the  component  being  considered 
may  not  be  appropriate.  If  so, 
leave  the  space  blank  on  the  form, 
or  line  through  it. 

As  appropriate  dimensions  are  chosen, 
enter  the  time  and  reliability  values 
by  dimension  on  the  form. 

Repeat  the  above  procedure  for  the 
mediating  process  and  output  aspects 
of  the  behavioral  step. 

In  the  event  that  more  than  one  Input 
or  Output  is  associated  with  a  par¬ 
ticular  step,  the  additional  compo¬ 
nents  are  evaluated  in  the  adjoining 
column . 
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SCORE  EVALUATION  SHEETS 


Separate  scores  are  computed  for 
each  component  of  each  step.  The  step 
scores  are  then  computed  using  the  com¬ 
ponent  scores. 

For  each  component,  add  the  Base 
Time  and  other  time  data  entered. 

Put  the  Total  in  the  appropriate 
space. 

To  determine  component  Reliability, 
multiply  together  all  reliability 
measures  entered  in  the  component. 
This  will  require  a  standard  desk 
calcu lator. 

Enter  the  fins,'  product  as  the 
Component  total  reliability. 

Repeat  this  process  for  all  compo¬ 
nents  in  the  step. 

Total  STEP  Time  is  simply  the  addi¬ 
tion  of  all  component  times  within 
the  step.  Enter  this  sum  in  the 
appropriate  STEP  TOTAL  space. 

Total  STEP  Reliability  is  the  pro¬ 
duct  of  multiplying  together  all 
the  component  reliabilities.  Enter 
this  result  in  the  appropriate  STEP 
TOTAL  space. 

Time  required  for  either  perceptual 
or  location  shift  is  assessed  when 
there  is  significant  time  lapse 
involved.  Enter  this  time  in  the 
appropriate  space.  Equipment  delays 
should  also  be  noted  at  this  point. 
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SUMMARIZE  RESULTS  BY  MISSION  AND  PHASE 


1.  Enter  identification  information  in  the  spaces  provided  at  the  bottom 
of  the  Summary  Form  (page  2  of  the  Appendix). 

2.  Arrange  completed  evaluation  sheets  by  phases  of  operation,  and  iden¬ 
tify  phases  on  the  Form  in  the  appropriate  places. 

3.  For  each  phase,  ADD  all  of  the  Input  component  t imes  from  the  evalua¬ 
tion  sheets.  Enter  the  sum  in  the  Input  column  under  time  for  the 
appropriate  phase. 

k.  Repeat  this  process  for'  the  Mediating  Process  and  Output  aspects 
that  occur  in  each  phase. 

5.  For  each  phase,  enter  the  sum  of  all  location  and  perceptual  shift 
times  and  equipment  delay  times  in  the  appropriate  space. 

6.  For  each  phase,  MULT IPLY  together  all  of  the  Input  component  rel la¬ 
bilities  from  the  evaluation  sheets.  Enter  the  product  in  the  Input 
column  under  "Reli."  for  the  appropriate  phase. 

7.  Repeat  this  process  for  the  Mediating  Process  and  Output  aspects 
that  occur  in  each  phase. 

FOR  TOTAL  PHASE  SCORES 

l.  For  each  phase,  ADD  the  Input,  Mediating  Process,  Output,  and  Per¬ 
ceptual  and  Location  Shift  t imes  and  note  the  total  phase  time  in 
the  space  provided. 

2.  For  each  phase,  MULTIPLY  together  the  Input,  Mediating  Process,  and 
Output  reliabilities  and  record  the  product  in  the  appropriate  space. 

FOR  TOTAL  ASPECT  SCORES 

1.  ADD  al I  of  the  INPUT  time  scores  for  a 1 1  phases.  Enter  this  as  the 
total  time  for  the  Input  aspect. 

2.  MULTIPLY  al 1  of  the  INPUT  rel iab i 1  it ies  for  all  phases.  Enter  the 
product  as  the  total  reliability  for  the  Input  aspect. 

3.  Repeat  'rtis  process  for  the  Mediating  Process  and  Output  aspects, 
recording  the  total  in  the  appropriate  spaces. 
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k.  ADD  a  1 1  of  the  perceptual  and  location  shift  times  and  the  equipment 
delay  times  for  all  phases,  recording  the  results  as  if  they  were 
aspect  total  scores. 

FOR  TOTAL  MISSION  SCORE 

l.  ADD  all  of  the  Total  Phase  Time  Scores.  The.  result  is  Total  Mission 
Time. 

2.  MULTIPLY  all  of  the  Total  Phase  Reliability  scores  together.  The 
result  is  the  Total  Mission  Reliability. 


Note:  These  scores  can  be  checked  in  the  following  way: 

Add  together  all  of  the  Aspect  Total  Time  scores  (except  for 
equipment  delay  time).  The  result  should  be  equal  to  Total 
Mission  Time.  Multiply  together  all  of  the  Aspect  Total 
Reliability  Scores.  The  result  should  be  equal  to  Total 
Mission  Reliability,  within  the  limits  of  rounding  error. 


SUMMARIZE  RESULTS  BY  COMPONENT 


1.  Enter  identifying  information  at  the  top  of  the  form.  A 
separate  Summary  Form  is  prepared  for  each  phase  of  operation. 

2.  From  the  Evaluation  Sheets,  observe  the  Input  component  for 
the  first  step.  Locate  this  component  on  the  Component  Sum¬ 
mary  Form.  Enter  the  step  number,  the  component  total  time 
and  total  reliability  in  the  space  provided. 

3.  Repeat  this  process  for  the  Mediating  Process  and  Output 
components  for  the  first  step.  Using  the  example  from  page 
14,  the  Component  Summary  Form  entries  would  appear  as  below. 

4.  Repeat  this  process  for  every  step  of  the  phase,  and  on 
additional  forms,  for  every  phase  of  the  mission. 

FOR  COMPONENT  TOTALS: 

1.  For  each  component,  ADD  all  time  scores.  Enter  this  sum  as 
the  Total  Time  for  that  component  at  the  bottom  of  the  fo-m. 
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2.  For  each  component,  MULTIPLY  together  all  reliability  scores. 
The  product  is  the  Total  Reliability  for  that  component,  and 
is  entered  in  the  appropriate  space  at  the  bottom  of  the  form. 

FOR  HISS  ION  COMPONENT  TOTALS: 

1.  ADD  each  of  the  component  Total  Time  scores  across  all  phases. 
The  result  is  the  total  time  required  for  this  component  dur¬ 
ing  the  entire  mission. 

2.  MULTIPLY  each  of  the  component  Total  Reliability  scores  over 
all  phases.  The  result  is  the  total  reliability  attributable 
to  the  individual  components  during  the  entire  mission. 
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DERIVE  RECOMMENDATIONS 


Redes i gn 

Redesign  recommendations  will  be  made  at  the  component  level  to 
overcome  design  problems  that  have  been  identified  in  the  course  of  the 
evaluation.  Potential  design  problems  may  result  from  two  different 
sources. 

1.  A  component  may  be  penalized  for  design  defects  with  a 
consequent  increase  in  time  requirements  or  lowering  of 
reliability.  In  some  cases,  however,  it  may  be  that 
redesign  to  decrease  time  may  result  in  a  decrease  in 
reliability,  or  vice  versa.  In  such  cases,  a  trade¬ 
off  in  design  must  be  made  based  on  the  mission  of  the 
system,  i.e.,  is  time  or  reliability  of  primary  impor¬ 
tance  in  meeting  mission  objectives. 

2.  A  wel I -des i gned  component  may  be  penalized  simply  be¬ 
cause  the  component  is  involved  in  a  number  of  steps. 

This  is  simply  a  result  of  the  fact  that  the  more 
often  a  step  is  performed,  the  more  total  time  it 
takes,  and  the  more  likely  it  is  that  an  error  will 
occur.  Even  though  the  component  is  we  I  I -des i gned , 
consideration  should  be  given  to  improvement  of  the 
design,  since  an  improvement,  even  though  slight  for 
a  single  step,  becomes  significant  over  a  large  num¬ 
ber  of  repet i t ions . 

The  following  procedure  is  suggested  for  generating  redesign  recom¬ 
mendations  to  insure  that  no  reasonable  possibilities  are  overlooked. 

1.  Analyze  Total  Component  Scores  on  the  Component  Sum¬ 
mary  Form. 

2.  Identify  those  components  that  contribute  most  signi¬ 
ficantly  to  Total  Mission  time  and  reliability. 

3.  Compare  the  individual  step  scores  vithin  the  selected 
components  with  the  base  time  and  optimum  reliability 
indicated  at  the  top  of  each  column. 
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4.  Select  for  further  consideration  those  components  where 
step  scores  are  generally  higher  than  base  time  or  lower 
than  optimum  reliability. 

5.  Compare  the  evaluation  of  each  component,  wherever  it 
occurs,  with  the  Data  Store  information  for  that  com¬ 
ponent. 

6.  Identify  alternative  dimensions  which  would  reduce 
time  and/or  increase  reliability. 

7.  Determine  the  total  amount  of  time  which  would  be  saved, 
or  the  increase  in  reliability  that  would  be  obtained 
for  each  potential  change  in  parameter  dimensions. 

Time  saved  is  the  difference  between  the  actual  time 
added  for  the  dimension  already  included  in  the  com¬ 
ponent  and  the  time  that  would  be  added  if  the  alter¬ 
nate  dimension  has  been  chosen  instead.  This  difference 
is  added  across  all  occurrences  of  the  component. 

8.  Recommend  that  the  component  be  redesigned  if  the  savings 
in  time  or  increases  in  reliability  are  sufficiently 
large.  Such  recommendations  must  be  feasible  within 
engineering  constraints. 

Tra i ni ng 

The  purpose  of  training  recommendations  based  on  interpretations 
the  Index  it  to  overcome  design  problems  of  the  equipment  that  are  not 
suitable  or  amenable  to  redesign,  it  is  not  intended  that  these  recom¬ 
mendations  include  overall  training  program  recommendatir ns.  Rather, 
they  should  be  restricted  to  training  suggestions  for  particular  compo¬ 
nents.  Two  types  of  training  recommendations  can  be  derived  from  the 
I ndex . 

1.  Those  steps  involving  difficult  components  can  be  speci¬ 
fied  for  specific  rehearsal  during  practice  sessions 
with  the  equipment. 

2.  The  specific  parameters  and  dimensions  of  a  component 
that  may  cause  operational  problems  can  be  identified 
for  emphasis  during  all  stages  of  training. 


The  following  procedure  is  suggested  for  the  derivation  of  training 
recommendations.  It  should  be  pointed  out,  however,  that  this  is  essen¬ 
tially  a  judgmental  process  by  the  evaluator  and  that  it  is  unlikely 
that  any  two  evaluators  would  arrive  at  exactly  matching  recommendations. 

1.  Identify  those  components  that  contribute  significantly 
to  total  mission  scores  that  are  not  subject  to  redesign. 

This  can  be  accomplished  at  the  same  time  that  redesign 
recommendations  are  being  considered. 

2.  Determine  the  type  of  recommends  ion  required  by  analyz¬ 
ing  the  nature  of  the  component. 

3.  Specify  the  information  about  the  component  required  to 
adequately  implement  the  recommendation,  e.g.,  the  spe¬ 
cific  dimensions  that  account  for  the  problem. 

Select  ion 

Selection  recommendations  based  on  Index  results  muse,  of  necessity, 
be  made  at  a  generalized  level.  This  is  due  to  the  fact  that  the  specific 
relationship  between  Index  results  and  selection  criteria  is  not  presently 
known.  However,  it  appears  reasonable  that  a  relationship  exists  between 
Aspects  of  behat ,or  and  general  classes  of  aptitudes  and  abilities.  The 
following  relationships  between  the  two  sets  of  variables  appear  to  be 
reasonable. 


Aspects 


Apt i tudes 


I  nputs 


Mediating  Processes 


Outputs 


Visua I  or  Aud i tory 
Recept ion 

Discrimination,  Decision 
Making,  Intelligence 

Motor  or  Verbal  Skills 


In  some  cases,  select, on  recommendations  may  be  made  as  alternatives 
to  training  recommendations.  For  example,  it  may  appear  to  the  evaluator 
•hat  providing  the  kind  of  training  required  to  insure  adequate  perfor¬ 
mance  would  be  difficult  if  not  impossible  to  accomplish,  whereas  select¬ 
ing  operators  with  the  required  skills  "built  in"  might  be  quite  feasible 
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The  following  general  procedure  is  suggested  for  generating  selection 
recommendations. 

1.  Analyze  the  ASPECT  Total  scores  on  the  Mission  Summary 
Form. 

2.  Select  the  Aspect  which  contributes  most  to  the  total 
mission  scores. 

3.  Check  the  individual  step  entries  for  the  Aspect  selected 
and  the  Component  Summaiy  data,  and  determine  the  general 
nature  of  the  problem. 

4.  Recommend  the  selection  of  operators  with  abilities  which 
will  overcome  or  minimize  the  problems  associated  with 
this  Aspect. 
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JUSTIFICATION 


All  design  recommendat ions  may  be  justified  by  demonstrating,  in 
specific  quantitative  terms,  the  extent  to  which  performance  time  and/or 
operator  reliability  will  be  enhanced  by  implementation  of  the  suggested 
modification.  For  example,  a  certain  design  change  may  reduce  perfor" 
mance  time  by  3-67  seconds,  and  increase  operator  reliability  from  .8732 
to  ,8961.  Justification  of  a  design  modification  usually  should  be 
expressed  in  terms  of  potential  changes  in  total  mission  scores.  How¬ 
ever,  computation  of  these  factors  should  be  possible  at  any  level. 

That  is,  revised  time  and  reliability  scores  may  be  stated  for  steps, 
components,  aspects,  or  any  other  phase  of  operation. 

Time 

Calculation  of  the  time  which  would  be  saved  by  implementation  of  a 
given  design  recommendation  is  easily  computed  with  the  following  formula: 

ST  -  <te  -  V* 

where: 

$.j.  *  time  saved  over  mission  performance, 

T^  ■  time  added  for  a  dimension  assessed  on  the  evaluation, 

T'n  ■  time  added  for  the  dimension  recommended  as  a  modification,  and 

N  ■  number  of  times  the  modified  componrnt  is  used  during  system 
operation. 

For  example,  assume  a  total  mission  time  of  270  seconds.  A  part  of 
this  total,  70  seconds,  is  attributable  to  operating  rotary  selector 
switches  10  times  (N).  One  relevant  factor  affecting  the  operating  time 
of  these  selector  switches  is  15°  distance  between  adjacent  positions. 

This  distance  adds  .60  seconds  (T^)  to  each  operation,  or  a  total  of  6.00 
seconds  for  all  10  operations.  A  recommendation  to  increase  distance 
between  adjacent  switch  positions  from  15°  to  25°  would  be  in  order. 
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Time  added  by  this  recommended  distance  is  0  seconds  (T^) .  Thus,  time 
saved  (Sy)  by  implementing  this  recommendation  would  be: 

Sy  =  (.60-0)  (10)  =6.00  seconds 

This  reduces  time  attributable  to  selector  switches  from  70  seconds  to 
64  seconds,  and  reduces  total  mission  time  from  270  to  264  seconds. 


Rel iabi  1  i tv 

The  tota 1  rel iabi 1 ity  estimates  are  products  of  a! 1  relevant  rel i- 
ability  scores  at  the  dimension  level.  Total  mission  reliability  is, 
thus,  the  product  of  a  1 1  dimens  ion  reliability  scores,  while  phase 
reliability  estimates  are  products  of  those  dimensions  contained  within 
the  phase.  Recommendations  which  change  a  dimension  also  change  the 
associated  reliability  score.  This  change  in  score  affects  the  total 
reliability  estimate.  It  is  this  effect  which,  with  time,  justifies 
the  recommendation.  The  determination  of  total  effect  becomes  more 
complex  as  the  number  of  changes  recommended  increases. 


Assuming  a  recommendation  results  in  a  single  different  dimension 
reliability  score,  the  new  or  recommended  total  reliability  (Rr)  can  be 
simply  determined.  The  procedure  is  to  divide  the  original  total  reli¬ 
ability  (Rq)  by  the  original  dimension  reliability  score  (rQ).  and  then 
multiply  this  quotient  by  the  new  or  recommended  dimension  reliability 
score  (>'r).  Thus: 


R 


r 


(rr)  or,  Rr 


However,  if  this  single  dimension  occurs  a  number  of  times,  then  the 
effect  each  time  It  occurs  must  considered.  The  above  formula  then 
becomes: 


R 


r 


R 

0 


n 


where  n  is  the  number  of  steps  affected  by  the  recommendation. 
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Even  with  n  quite  small,  the  computation  becomes  laborious.  To 

rr  n 

reduce  the  effort  involved,  expected  values  of  (— )  have  been  computed 

o 

and  are  presented  in  Table  I.  The  table  presents  all  quotients  from 
qggq  9999 

gyOQ  to  which  exhausts  the  range  of  values  expected,  raised  to 

powers  of  n  sufficient  to  cover  most  evaluations.  For  further  conven¬ 
ience,  the  quotients  from  the  above  range  are  presented  in  Table  II. 
This  ♦'able  presents  the  range  of  values  of  r  as  dividend  and  divisor. 
The  intersecting  cell  in  the  table  presents  the  quotient. 


The  procedure  for  determining  the  effects  of  a  recommendation  on 
total  reliability  are  as  follows: 


1. 

2. 


3. 


4. 


r  n 

Enter  the  proper  values  in  the  formula  R  =  R  (— ) 
r  r  r  o  r 

o 

r 

To  determine  the  quantity  (-p — )  ,  ENTER  TAbLE  II  with  the  two 

o 

quantities,  and  read  the  quotient  from  the  intersecting  cell, 

or  perform  the  division  in  the  conventionai  way. 

r  n 

To  determine  the  quantity  (“)  ,  ENTER  TABLE  I  with  the  quo- 

o 

tient  from  (2)  above,  for  the  value  of  n  required. 

Multiply  this  figure  by  Rq.  The  result  is  the  total  reli¬ 
ability  assuming  the  recommendation  is  implemented  (Rr) . 

This  total  reliability  may  be  total  component,  total  step, 
total  phase,  or  total  mission  reliability,  depending  upon  the 
needs  of  the  specific  case. 


Note:  Although  it  is  unlikely,  some  relevant  values  of  r  or 
n  may  not  be  included  in  the  tables.  In  this  event,  conven¬ 
tional  computation  routines  will  be  required. 


25 


Determination  of  Joint  Recommendations 

To  determine  the  effects  of  a  1 1  redesign  recommendat i ons  (or  for 
any  combination  of  recommendations)  on  total  mission  reliability: 


n  n  n  n 


n  n  n  n 


where  MR^  is  the  Mission  Reliability  given  all  redesign  recommendations, 
and  MRq  is  the  original  Mission  Reliability  and  the  other  quantities  as 
identified  previously. 

Since  the  previous  steps  of  the  preceding  procedure  have  identified 
r  n 

each  of  the  ( — )  values,  the  above  formula  may  be  written  in  the  follmt-'- 
o 

ing  form  for  computational  convenience: 
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SUMMARY  RECOMMENDATION  TABLE 


A  form  presented  in  the  Appendix  is  provided  with  the  Index  materials 
for  summarizing  all  the  recommendations  made.  The  process  involved  is 
simply  collecting  and  briefly  expressing  the  recommendations  made.  This 
form  is  intended  to  present,  in  a  single  location,  the  following  infor¬ 
mation: 

1.  Brief  descriptions  of  redesign  recommendat ions  for  all 
relevant  components,  and  the  number  of  operating  steps 
affected  by  the  recommendations. 

2.  Time  and  reliability  estimates  for  both  existing  and 
redesign  configurations.  This  data  is  presented  at 

the  component  phase  and  total  mission  level. 

1 

3.  Brief  resumes  of  selection  and  training  recommendations 
made.  The  justifications  here  are  logical,  ratner  than 
quantitative  and,  thus,  are  usually  more  difficult  to 
summarize.  If  a  clear  summary  justification  cannot  be 
presented,  there  should  be  a  reference  to  the  logical 
development  of  the  recommendation  in  the  text. 
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APPENDIX 


EVALUATION  SHEET 
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MISSION  AND  PHASE  SUMMARY  FORM 


Percept . 
and  Loc. 
Time 

1 

1 

| 

1 

1 

PUTS 

Re  1  i  . 

■ 

1 

■ 

OUT 

Time 

1 

1 

co  — 
o  — 

OC  0) 
cl  a: 

1 

1 

MED. 

Time 

■ 

■ 

PUTS 

Rel  i  . 

1 

1 

IN 

Time 

1 

1 

Phase  Scores 

C4 

J- 

vO 

< 

H 

O 


</> 

UJ 

ee 

o 

o 

l/l 

U 

UJ 

CL 

V> 

< 


c 

i 

c 

o 

a 

•— * 

1/1 

3 

CT 

a* 

•  — 

UJ 

x: 

Evaluator 


MEDIATING 


